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Current  research  in  cognitive  and  developmental  psychology  can 
influence  the  development  of  new  psychometric  methodology.  Research 
on  the  diagnosis  of  performance  regularities  at  different  levels  of 
learning  and  development  should  contribute  to  assessment  measures  that 
provide  information  useful  for  instructional  decision  and  guidance. 
Investigation  of  the  structure  and  cognitive  processes  of  highly 
competent  performers  in  various  knowledge  domains  is  beginning  to 
suggest  techniques  for  assessing  the  attainment  of  increasingly 
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The  Future  of  Testing:  A  Research  Agenda  for 
Cognitive  Psychology  and  Psychometrics* 

Robert  Glaser 
University  of  Pittsburgh 


It  is  becoming  less  than  constructive  to  recount  the  trials  of 
the  psychometric  enterprise  as  it  relates  to  the  uses  of  tests  in 
education.  We  are  all  familiar,  to  soue  degree,  with  the  thoughtful 
writing  and  the  rhetoric  on  such  issues  as  test  bias,  tests  as 
selective  and  exclusionary  instruments,  the  questioned  value  of 
intelligence  tests  as  assessments  of  learning  potential,  the 
reliability  of  classification  and  Instructional  assignment  through 
tests,  and  the  difficulty  of  setting  appropriate  standards  for 
achievement  and  competence  assessment.  Pros  and  cons  are  expressed 
with  intellectual  understanding,  ignorance,  social  justification,  and 
passion  by  concerned  citizens  and  parents,  teacher  organizations, 
psychometricians  and  psychologists  of  all  kinds,  and  consumer  advocacy 
groups.  The  malaise  and  multilogue  continues,  and  what  we  must  do  as 
best  we  can  is  to  analyze  and  understand  the  deep  structure  and  roots 
of  this  unrest. 


*Preparation  of  this  paper  was  sponsored  by  the  Personnel  and 
Training  Research  Programs,  Psychological  Sciences  Division,  Office  of 
Naval  Research,  and  by  the  Learning  Research  and  Development  Center, 
which  is  supported  in  part  by  funds  from  the  National  Institute  of 
Education,  U.  S.  Department  of  Education.  A  portion  of  this  paper 
was  presented  at  the  annual  meeting  of  the  American  Psychological 
Association  in  Montreal,  September  1980  as  a  presidential  address 
given  by  the  author  to  the  APA  Division  of  Evaluation  and  Measurement. 
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Rather  than  examining,  and  thereby  entering,  the  debate  where  all 
sides  can  be  both  praised  and  faulted  in  different  respects,  it  seems 
reasonable  to  me  to  reflect  on  the  scene  and  see  what  can  be  learned. 
I  shall  do  this  by  taking  a  futurist's  view,  and  I  see  this  not  as 
avoiding  the  debate,  but  hopefully  as  a  way  of  focusing  its 
directions.  Two  perspectives  seem  appropriate.  The  first  is  to 
examine  what  I  see  as  the  social  and  educational  demands  of  today  and 
tomorrow.  The  second  is  to  examine  relevant  research  on  human 
cognition  and  cognitive  development  in  order  to  speculate  on  a 
research  agenda  for  the  measurement  of  human  performance  that  can 
address  these  social  and  educational  requirements. 

Before  beginning  this  examination,  I  should  say,  as  a  general 
comment,  that  I  believe  that  the  general  uprising  of  litigation, 
social  questioning,  and  scientific  probing  will  have  lasting  benefits. 
First,  test  developers  and  test  users,  as  well  as  society  in  general, 
have  been  sensitized  to  how  apparently  useful  practices  may  actually 
perpetuate  discrimination  and  inequity.  Second,  in  the  face  of  the 
growth,  acceptance,  and  power  of  tests,  psychologists  and  educators 
have  been  alerted  to  the  fact  that,  like  other  professionals,  they 
will  be  held  responsible  for  their  conduct.  They  must  re-examine 
their  testing  practices  and  their  interpretations  of  tests  as 
professional  tools  that  contribute  to  educational  goals.  Third,  the 
attacks  on  testing  have  accelerated  technical  and  scientific 
examination  of  the  fundamental  assumptions  and  basic  knowledge  that 
underlie  test  design  and  various  assessment  procedures. 


Social  and  Educational  Demands 


New  developments  in  testing  procedures  for  the  purposes  of 
education  will  reflect  the  above  trends  as  researchers  and 
practitioners  respond  to  the  critical  social-educational  demands  of 
our  society.  1  compress  these  demands  into  three  categories:  the 
need  for  access  to  education,  the  requirement  for  competence,  and  the 
demand  for  equitable  improvement  in  the  skills  of  learning  that 
contribute  to  educational  attainment. 

Access  to  education.  As  is  well  known,  the  demand  for  tests 
arose  during  the  period  when  school  attendance  was  made  compulsary, 
and  when  higher  education  was  developing  its  strengths.  Educators 
faced  the  unprecedented  dilemma  of  dealing  with  the  range  and 
diversity  that  individuals  bring  to  schooling.  They  needed  ways  of 
determining  which  children  and  youths  would  be  able  to  profit  from 
some  form  of  instruction  as  given  in  ordinary  school  and  college 
practices  and  as  designed  essentially  for  the  majority  population. 
The  task  of  dealing  with  individual  diversity  was  handled  in  the 
context  of  the  social  climate  of  that  time,  the  current  forms  of 
school  organization,  and  the  emergent  psychometric  technology.  The 
struggle  of  educators  to  deal  with  diversity  was  beginning.  Selection 
testing  was  seen  as  part  of  the  solution  since  it  could  assist  in 
predicting  which  individuals  needed  special  education  or  could  be 
discouraged  from  further  schooling. 

At  the  present  time,  the  situation  has  changed:  All  children  and 
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youth  are  in  school,  albeit  with  significant  instances  of  exclusionary 
practices  and  dropout.  It  seems  clear  that  we  are  over  the  threshold 
in  the  transition  from  education  as  a  highly  selective  enterprise  to 
one  that  is  focused  upon  developing  an  entire  population  of  educated 
people.  A  selective  system  is  no  longer  the  prevalent  educational 
demand.  (Even  though  the  choice  to  be  selective  or  not 
remains.)  There  is  now  less  emphasis  on  selecting  individuals  for 
available  educational  opportunities ,  and  more  on  helping  them  succeed 
in  these  opportunities.  The  selective  emphasis  placed  too  much  of  the 
burden  for  learning  on  the  condition  of  the  student,  and  too  little 
burden  on  the  possible  influences  of  teaching,  training,  and 
instruction.  We  are  now  also  aware  that  we  have  not  come  close  to 
assessing  the  liir.t  “s  of  effective  instruction  and  that  we  need  to 
begin  to  do  this.  The  requirement  now  is  to  design  a  helping  society 
where  we  devise  means  for  providing  educational  opportunities  for  all 
in  equitable  ways. 


This 

shift  in 

emphasis 

poses 

problems  in  the  use 

of  tests 

for 

monitoring 

access 

to  education. 

Selection  devices 

used  at 

the 

university 

level  and  tests 

used 

to  make  assignment 

decisions 

in 

earlier  education  have  generally  not  been  designed  to  guide  specific 
instructional  practices.  They  have  rather  identified  those  who  are 
likely  to  be  successful  performers  (Astin,  1979).  As  Robert  Thorndike 
says  in  a  1975  essay  entitled  "Mr.  Binet' s  Test  70  Years  Later": 
Accepting  the  goal  of  maximizing  the  effectiveness  of 
education  for  all  children  and  youth,  we  must  face  up  to  the 
problem  that  we  have  long  acknowledged  but  seldom  dealt  with 
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effectively — the  problem  of  providing  for  each  individual 
the  educational  treatment  that  will  be  most  effective  in 
developing  that  person's  potential.  A  good  measure  of 
scholastic  aptitude  is  not  automatically  a  good  guide  to  the 
optimal  educational  treatment.  Binet's  test,  like  others 
used  in  education,  must  be  judged  in  terms  of  its  ability  to 
facilitate  constructive  adaptations  of  educational  programs 
for  individuals.  This  is  the  challenge  for  the  next  70 
years  .  ( p .  7 ) 

J.  McVicker  Hunt  in  the  same  year  expressed  this  condition  as 
follows : 

Psychological  assessment  should  guide  teaching.  It  should 
tell  a  teacher  what  kinds  of  assignment  and  curricular 
materials  a  given  child  can  utilize  profitably  to  foster  his 
psychological  development ... .The  form  of  psychological 
assessment  now  most  prevalent  in  education  fails  utterly  to 
do  this.  (p.  545) 

Thorndike  and  Hunt  are  individuals  with  original  ideas  but  they  would 
agree  that  their  statements  have  been  in  our  minds  for  a  long  time. 
However,  they,  like  others,  realize  that  such  statements  must  be 
reiterated  often  enough  until  they  engender  action. 

Teachers  and  schools  need  information  on  individuals  that  is 
oriented  toward  instructional  decision  rather  than  prediction.  Tests 
in  a  helping  society  are  not  mere  indexes  that  predict  that  the 
individual  child  will  adjust  to  the  school,  or  that  relieve  the  school 
from  assisting  the  student  to  achieve  as  much  as  possible.  The  test 


and  the  instructional  decision  should  be  an  integral  event. 
Separation  betweyi  a  test  that  monitors  access  to  education  and  the 
conditions  of  that  education  is  only  tolerable  for  those  institutions 
who  choose  to  say  this  is  the  way  we  teach,  and  our  plan  is  to  pick 
only  those  applicants  who  are  most  successful  with  our  methods. 

Test  developers  and  psychometric  scholars  have,  to  some  extent, 
put  their  minds  to  this  problem.  But  more  ingenuity,  disregard  of 
past  conceptions,  and  cooperation  with  and  by  teachers  and  schools  is 
required.  Reviews  of  the  effectiveness  of  diagnostic  test  batteries 
have  been  highly  critical  and  are  not  encouraging  (Coles,  1978; 
Ysseldyke ,  1979).  It  appears  that  tests  continue  the  tradition  of 
identifying  particular  symptoms  of  learning  disability  or  inexperience 
and  then  using  this  information  for  classification  and  differential 
assignment.  There  is  little  empirically  derived  and  conceptually 
understood  relationship  between  test  score  information  and  specific 
instructional  activity.  Overall,  I  see  little  significant  progress 
along  these  lines  unless  more  flexible  environments  are  introduced  in 
school  systems.  This  would  permit  differential  instructional 
practices  that  could  be  coordinated  with  useful  diagnostic  assessment 
so  that  the  testing  and  the  teaching  become  integral  events. 

The  requirement  for  competence .  A  second  aspect  of  the  current 
social  and  educational  climate  is  that  more  people  are  better  educated 
and  literacy  levels  are  higher  than  ever  before.  In  addition,  more 
people  are  demanding  that  they  attain  these  levels.  In  an  historical 
survey  of  the  nature  of  literacy,  Resnick  and  Resnick  (1977)  conclude 


the  following: 

Today,  the  term  "functional  literacy"  has  come  to  mean  the 
ability  to  read  common  texts  such  as  newspapers  and  manuals 
and  to  use  the  information  gained,  usually  to  secure 
employment.  The  objectives  of  functional  literacy  may  seem 
limited,  yet  this  mass-literacy  criterion  is  stronger  than 
that  of  any  earlier  period  of  history.  Achieving  universal 
literacy  as  it  is  now  defined  poses  a  challenge  not 
previously  faced.  We  estimate  that  literacy  standards  in 
the  United  States  in  the  1990's  will  be  both  more  demanding 
and  more  widely  applied  than  any  previous  standard. 

(P-  383) 

The  point  to  be  made  is  that  we  now  expect  individuals  to  not  only 
master  beginning  skills  but  to  also  be  able  to  learn  from  reading  and 
to  make  inferences  and  solve  problems  with  verbal  and  mathematical 
input.  In  many  subject  matter  domains,  attainment  of  comprehension 
and  problem  solving  skills  is  now  a  greater  problem  than  mastery  of 
elementary  skills. 

Related  to  the  demand  for  higher  levels  of  learning  there  looms 
on  the  horizon  the  press  for  education  in  the  skilled  trades,  the 
sciences  and  the  professions.  Higher  levels  of  competence  coupled 
with  productivity  and  accountability  are  absolutely  required  to  meet 
the  competition  from  increasingly  prosperous  nations  of  the  world. 
Competition  in  sports,  engineering  technology  in  electronics  and  steel 
production,  maintaining  leads  in  scientific  knowledge  and  theory, 
creative  writing  and  other  art  forms,  and  devising  systems  for 
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education  and  health  care,  requires  levels  of  skill  and  competence 
that  are  a  singular  challenge  for  our  educational  system  and  its 
related  technologies  such  as  testing. 

Skills  of  learning:  Intelligence  and  general  aptitude.  We  are 
now  well  aware  that  the  tests  of  general  intelligence  and  of  verbal 
and  quantitative  aptitude  in  use  in  our  educational  system  measure  the 
kind  of  intellectual  performance  called  "general  scholastic  ability." 
The  abilities  that  are  tested  are  those  that  correlational  evidence 
has  shown  to  be  predictive  of  success  in  school  learning.  These  tests 
are  -not  tests  of  intelligence  in  some  abstract  form.  Rather,  if  we 
base  our  conclusions  on  what  these  tests  measure,  in  terms  of  their 
most  effective  use — that  is,  their  predictive  validity — then  the 
verdict  is  that  they  are  primarily  tests  of  scholastic  aptitude  or 
scholastic  ability  (e.g.,  Cronbach,  1970;  Scarr,  1978;  Tyler,  1963). 
They  measure  certain  abilities  that  are  helpful  in  most  school  work  as 
it  is  usually  conducted  in  present-day  school  situations.  However, 
our  understanding  of  these  abilities  for  learning  is  very  incomplete. 
We  need  to  explain  the  correlational  and  factor  analytic  knowledge 
that  has  accumulated  in  psychometrics  in  order  to  be  able  to  enhance 
or  remediate  these  abilities  through  educational  and  other 
environmental  intervention.  Many  years  ago  an  eminent  psychometrician 
recognized  the  need  for  such  understanding.  In  1964,  Quinn  McNemar 
wrote : 

There  have  been  thousands  of  researches  on  the  multitudinous 

variations  from  organism  to  organism. .. .But  these  studies  of 

individual  differences  never  come  to  grips  with  the  process , 


or  operation,  by  which  a  given  organism  achieves  an 
intellectual  response.  Indeed,  it  is  difficult  to  see  how 
the  available  individual  difference  data  can  be  used  even  as 
a  starting  point  for  generating  a  theory  as  to  the  process 
nature  of  general  intelligence  or  of  any  other  specified 
ability,  (p.  881) 

We  are  now  beginning  to  systematically  assess  the  influences  of 
family,  preschool,  later  schooling,  and  social-cultural  surrounds  upon 
intellectual  processes  that  contribute  to  intelligence  and  aptitude. 
We  no  longer  overlook  the  fact  that  test  scores  are  related  to 
acquired  abilities,  life  experiences,  and  educational  opportunity.  In 
contrast  to  the  general  attitude  in  the  days  of  the  Army  Alpha,  and  in 
more  recent  years,  we  no  longer  underemphasize  the  fact  that  aptitude 
tests  give  weight  to  educational  and  social  advantage  or  disadvantage. 
As  one  consequence  of  this,  it  is  now  an  urgent  matter  that  we  attempt 
to  understand  the  cognitive  processes  that  are  involved  in  tests  of 
mental  abilities  (Carroll,  1978;  Sternberg,  1977),  and  begin  to  ask 
how  these  abilities  might  be  learned. 

I  have  described  three  aspects  of  the  current  social-educational 
Zeitgeist:  the  shift  from  a  selective  system  to  a  system  that  can  be 
helping,  adaptive,  and  instructionally  oriented;  the  necessity  for 
attaining  and  assessing  high  levels  of  competence;  and  the  presence 
of  a  social  attitude  more  willing  and  scientifically  able  to  unpack 
the  factors  of  mental  abilities  and  to  test  the  limits  of  their 
instructability.  New  thinking  and  research  and  development  in 
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testing,  as  they  are  Intertwined  with  the  goals  and  methods  of 
education,  will  need  to  be  responsive  to  these  pressures. 

A  Research  Agenda 

It  has  been  necessary  for  me  to  talk  about  social  and  educational 
climate,  because  I  believe  that  education  will  change  in  response  to 
these  conditions,  and  coordinate  adjustments  will  be  necessary  for 
testing  and  psychometrics.  These  adjustments  will  reflect  both  social 
and  scientific  developments,  and  it  is  to  the  latter  that  this  paper 
is  primarily  addressed.  In  what  follows,  I  review  some  recent 
scientific  developments  that  I  think  will  be  influencing  future  theory 
and  practices  in  instruction  and  testing.  In  particular,  I  discuss: 
the  diagnosis  of  level  of  performance,  the  nature  and  assessment  of 
competence,  and  the  understanding  and  improvement  of  aptitude  for 
learning.  The  recent  work  along  these  lines  suggests  concepts, 
techniques,  and  research  possibilities;  they  comprise  a  possible 
research  response  to  the  social-educational  pressures  that  I  have 
described . 

Diagnosis  of  Levels  of  Performance 


An  important  skill  of  teaching  is  the  ability  to  synthesize  from 
a  student's  performance  an  accurate  picture  of  a  student's 
misconceptions  that  lead  to  error.  This  task  goes  deeper  than 
identifying  incorrect  answers  and  pointing  these  out  to  the  student: 
It  should  identify  the  nature  of  the  concept  or  rule  that  the  student 
is  employing  that  governs  his  performance  in  some  systematic  way  (in 
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most  cases,  Che  student's  behavior  is  not  random  or  careless,  but  Is 
driven  by  some  underlying  misconceptions  or  by  Incomplete  knowledge) . 
Research  attempting  to  analyze  performance  in  this  way  has  been 
carried  out  in  arithmetic,  composition,  and  the  understanding  of 
scientific  concepts. 

Arithmetic  bugs .  In  work  that  combines  the  fields  of  artificial 
intelligence  and  cognitive  psychology,  Brown  and  Burton  (1978)  have 
developed  computer  models  of  procedural  skills.  They  use  a  technique 
called  "procedural  networks"  as  a  framework  for  constructing 
diagnostic  models — models  that  capture  a  student’s  misconceptions  or 
faulty  behavior.  Their  potentially  practical  modeling  technique 
provides  an  identification  of  mistakes  that  a  student  is  making  and  an 
explanation,  in  terms  of  faulty  procedures,  of  why  those  mistakes  are 
being  made.  Systems  of  this  kind,  computerized  or  not,  have  important 
implications  for  conceptions  of  testing  because  students  are  evaluated 
not  on  the  number  of  errors  appearing  in  their  tests,  but  rather  on 
the  basis  of  the  fundamental  misconceptions  that  influence  their 
performance . 

Consider  the  Brown  and  Burton  example  in  diagnosing  arithmetic 
skills.  Here  are  five  items  that  sample  a  student's  performance  doing 
addition: 


41 

328 

989 

66 

216 

+9 

+917 

+52 

+887 

+13 

50 

1345 

1141 

1053 

229 

The  task  is  to  discover,  by  examining  these  items,  the  student's  bug. 
Once  you  have  discovered  the  bug,  test  your  hypothesis  by  simulating 
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the  student's  performance  so  as  to  predict  his  results  on  subsequent 
test  problems  such  as  the  following: 

446  201 

+815  +399 

The  bug  here  is  simple:  The  student,  after  determining  that 

carrying  is  necessary,  accumulates  the  amount  across  columns.  For 
example,  in  the  student's  second  problem,  he  proceeds  as  follows: 
8  +  7  »  15;  he  writes  5  and  carries  1;  2+1*3  plus  the  1  carry  is 

4;  in  the  last  column,  3+9-12  but  the  1  carry  from  the  first 
column  is  still  there,  so  adding  it  to  this  column  gives  13,  with  the 
answer  1,345.  If  this  is  the  bug,  then  the  answer  to  the  test 
problems  will  be  1,361  and  700.  Brown  and  Burton  point  out  that  this 
bug  may  not  be  so  absurd  if  one  considers  that  children  might  use 
their  fingers  to  remember  the  carry  and  then  forget  to  bend  them  back 
after  each  carry  is  added.  Some  other  examples  from  subtraction  are 
shown  in  Figure  1. 


Consider  the  following  case  where  a  student  proceeded  through  a 
portion  of  the  school  year  with  the  teacher  thinking  that  he  was 
exhibiting  random  behavior  in  his  arithmetic  performance.  The  sample 
of  the  student's  work  is  the  following: 

7  9  8  19  87 
+8  +5  +3  +4  +93 
15  14  11  23  11 


365  679  27,493 

+574  +794  +1 , 509 

819  111  28,991 


The  student  subtracts  the  smaller  digit  in  each  column  from  the  larger  digit 
regardless  of  which  is  on  top. 


143 
-  28 


155 


143 
-  28 
125 


1300 

-522 

“575 


140 
-  21 


140 


1300 

-522 

188 


When  the  student  needs  to  borrow,  he  adds  10  to  the  top  digit  of  fhe  current 
column  with  out  subtracting  1  from  the  next  column  to  the  left. 


When  borrowing  from  a  column  whose  top  digit  is  0,  the  student  writes  9  but 
does  not  continue  borrowing  from  the  column  to  the  left  of  the  0. 


Whenever  the  top  digit  in  a  column  is  0,  the  student  writes  the  bottom  digit  in 
the  answer;  i.e.,  0  —  N  =  N. 


Whenever  the  top  digit  in  a  column  is  0,  the  student  writes  0  in  the  answer,  i.e., 
0  —  N  °  0. 


When  borrowing  from  a  column  where  the  top  digit  is  0,  the  student  borrows 
from  the  next  column  to  the  left  correctly  but  writes  10  instead  of  9  in  this 
column. 

When  borrowing  into  a  column  whose  top  digit  is  1 ,  the  student  gets  10  instead 
of  11. 


662  Once  the  student  needs  to  borrow  from  a  column,  s/he  continues  to  borrow 
-357  from  every  column  whether  s/he  needs  to  or  not. 

205 

662  The  student  always  subtracts  all  borrows  from  leftmost  digit  in  the  top  number. 
-357 


Figure  1.  Manifestations  of  some  subtraction  bugs  (from  Brown  &  Burton  1978). 
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In  this  set  of  items,  the  clue  to  the  student's  faulty  procedures  is 
to  be  seen  in  the  number  of  ones  in  his  answers.  Every  time  the 
addition  of  the  column  involves  a  carry,  a  1  appears  in  the  column; 
he  is  writing  down  the  carry  digit  and  forgetting  about  the  units 
digit.  He  is  getting  the  simple  additions  correct  but  adopts  a 
different  and  incorrect  procedure  when  presented  with  two-  and 
three-place  numbers. 

The  development  of  such  a  general  diagnostic  procedure  begins 
with  the  decomposition  of  a  complex  skill  into  component  procedures 
that  contain  the  elements  of  the  underlying  ability,  for  example, 
recognizing  a  digit,  writing  a  digit,  working  right  to  left  by  one 
method  or  left  to  right  by  another  method,  carrying  onto  the  top 
operand,  carrying  into  the  answer,  and  so  forth.  A  computational 
procedure  employing  these  components  can  be  developed  to  perform  the 
skill,  and  a  diagnostic  model  is  then  defined  as  a  way  of  representing 
a  student's  procedural  knowledge  as  some  variant  of  the  correct 
version  of  this  procedure.  Misconceptions  are  identified,  through  the 
student's  patterns  of  error,  that  result  from  incorrect 
implementations  of  the  various  component  skills. 

Diagnosing  performance  in  this  way  would  be  useful  for  a  variety 
of  pedagogical  purposes.  Brown  and  Burton  (1978)  suggest  that  in 
addition  to  its  use  for  diagnostic  teaching  purposes,  it  can  be  used 
in  teacher  training.  It  can  encourage  the  teacher  to  see  that  the 
apparently  random,  careless,  or  lazy  behavior  of  a  student  is 
frequently  a  rather  complex  and  logical  thought  process  toward  which 
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teaching  can  be  directed.  A  diagnostic  emphasis  of  this  kind  Is  quite 
Interesting  and  impressive  to  teachers;  they  view  it  has  an  important 
aspect  of  their  own  skills  and  as  a  way  of  respecting  the  systematic 
intelligence  of  their  students.  Such  a  system  of  procedural  analysis 
might  also  be  a  methodological  tool  for  determining  how  well  a 
particular  test  or  set  of  items  can  discover  and  delineate  common 
misconceptions  that  occur  in  the  course  of  learning. 

Writing  skill.  Somewhat  analogous  to  the  identification  of 
systematic  misconceptions  in  arithmetic  that  I  have  just  described  is 
recent  work  on  the  study  of  error  in  writing  and  composition 
<Bartholomae ,  1980;  Hayes  &  Flower,  1980;  Shaughnessy,  1977a, 
1977b).  There  has  been  much  discussion,  by  teachers  of  basic  writing 
courses,  about  how  the  pedagogy  of  such  courses  could  be  improved,  and 
about  how  systematic  inquiry  can  be  undertaken  into  the  nature  of  the 
writing  skill  that  students  bring  to  their  classes,  and  how  those 
skills  develop  into  improved  writing. 

The  helping  attitude  in  education  that  I  described  earlier  is 
clearly  reflected  in  a  leading  book  in  this  area — a  book  by 
Shaughnessy  (1977b)  that  reflects  her  experiences  in  teaching  writing 
during  the  years  when  the  City  University  of  New  York  adopted  an 
admissions  policy  that  guaranteed  to  every  city  resident  with  a  high 
school  diploma  a  place  in  one  of  its  tuition  free  colleges. 
Shaughnessy  writes: 

Those  pedagogies  that  served  the  profession  for  years  seem 

no  longer  appropriate  to  large  numbers  of  our  students,  and 
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their  inappropriateness  lies  largely  in  the  fact  that  many 
of  our  students .. .are  adult  beginners  and  depend  as  students 


did  not 

depend  in 

the  past  upon 

the 

classroom 

and  the 

teacher 

for  the 

acquisition  of 

the 

skill  of 

writing. 

(P-  317) 

This  requirement  has  generated  research  of  an  ambitious 
kind — study  of  the  nature  of  systematic  error  and  the  changes  in  this 
performance  as  writing  skill  develops.  As  in  the  mathematics  work, 
errors  are  not  seen  as  random  guesses,  error  is  seen  in  a  rich, 
compassionate,  and  instructive  way.  Poor  performance,  which  tends  to 
bring  feelings  of  fear  and  vulnerability  to  the  student  and 
frustration  and  despair  to  the  teacher,  is  now  seen  as  a  form  of 
systematic  intellectual  performance  that  provides  a  form  of 
information  to  the  student  and  to  the  teacher  as  a  basis  for 
development . 

In  this  work,  there  is  little  of  the  neat  precision  that  is 
possible  in  the  analysis  of  the  arithmetic  performance  that  I 
described.  Yet  error  analysis  in  writing  can  also  be  a  useful  method 
of  diagnosing  the  performance  of  a  writer  to  discover  the  rules  the 
writer  employs  that  produce  mistakes.  Errors  fall  into  three  general 
categories:  errors  that  are  evidence  of  intermediate,  transitional 
rules  for  writing  prose,  approximating  the  standard  rules;  errors 
that  are  slips  of  the  pen  as  a  writer's  mind  rushes  ahead  faster  than 
his  or  her  hand;  and,  errors  of  dialect  interference,  where  the 
writer  use  idioms  and  syntax  from  a  spoken  language. 
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"The  error  analyst  is  primarily  concerned .. .with  errors  that  are 
evidence  of  some  intermediate  system.  This  kind  of  error  occurs 
because  the  writer  is  an  active,  competent  language  user  who  uses  his 
knowledge  that  language  is  rule-governed. . .to  construct  hypotheses 
chat  can  make  an  irregular  or  unfamiliar  language  more  manageable. 
The  problem  comes  when  the  rule  is  incorrect  or,  more  properly,  when 
it  is  idiosyncractic,  belonging  only  to  the  language  of  this  writer” 
(Bartholomae ,  1980,  p.  9).  Bartholomae' s  work  on  error  analysis  with 
remedial  entering  college  students  has  demonstrated  that  the  errors  in 
the  writing  performance  of  these  writers  evidences  not  a  general  lack 
of  linguistic  competence,  but  an  idiosyncractic  use  of  language  that 
can  be  analyzed  and  categorized  to  reveal  an  "inter language,"  the 
rules  the  writer  uses  to  record  meaning  on  paper.  The  myriad  errors 
the  inadequate  writer  makes  in  producing  prose  are  traced  by  the  error 
analyst  to  the  rules  of  this  interlanguage  or  intermediate  system  that 
is  the  writer's  approximation  to  the  standard  idiom. 

To  give  an  example,  Bartholomae  (1980)  examines  the  errors  in  one’ 
student's  two  page  paper.  It  is  written  by  a  student,  John,  in 
response  to  an  assignment  to  review  earlier  papers  he  had  written  on 
significant  moments  in  his  life,  and  now  his  task  is  to  write  a  paper 
on  the  general  question  of  how  and  why  people  change: 

This  assignment  call  on  chosing  one  of  my  incident  making  a 
last  draft  out  of  it.  I  found  this  very  differcult  because 
I  like  tham  all  but  you  said  I  had  to  pick  one  so  the  Second 
incident  was  decide.  Because  this  one  had  the  most 


important  insight  to  my  life  that  I  indeed  learn  from.  This 
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insight  explain  why  adulthood  mean  that  much  as  it  dose  to 
me  because  I  think  it  alway  influence  me  to  change  and  my 
outlook  on  certain  thing  like  my  point-of-view  I  have  one 
day  and  it  might  change  the  next  week  on  the  same  issue.  So 
in  these  frew  words  I  going  to  write  about  the  incident  now. 

My  exprience  took  place  in  my  high  school  and  the  reason  was 
out  side  of  school  but  I  will  show  you  the  connection.  The 
sitution  took  place  cause  of  the  type  of  school  I  went  too. 

Let  me  tell  you  about  the  sitution  first  of  all  what  happen 

* 

was  that  I  got  suspense  from  school.  For  thing  that  I  fell 
was  out  of  my  control  sometime,  but  it  taught  me  alot  about 
respondability  of  a  growing  man.  The  school  suspense  me  for 
being  late  ten  time.  I  had  accummate  ten  dementic  and  had 
to  bring  my  mother  to  school  to  talk  to  a  conselcr  and 
Prinpicable  of  the  school  what  when  on  at  the  meet  took  me 
out  mentally  period. 

Bartholomae  suggests  that  in  response  to  this  paper  an  instructor 
could  come  to  the  conclusion  that  John  cannot  write  and  send  him  off 
to  a  workbook  to  learn  how  to  write  correct  sentences  and  then  come 
back  to  actual  writing.  A  more  analytical  instructor  could  chart 
patterns  of  error  in  this  text.  Of  the  approximately  40  errors  in  the 
first  200  words,  the  majority  fall  under  fairly  specific  categories: 
verb  endings,  noun  plurals,  syntax,  and  spelling.  Instruction  for  the 
student  could  be  focused  on  these  specific  kinds  of  errors.  For 
example,  the  verb  endings  almost  all  involve  "s"  or  ”ed"  endings  which 
could  Indicate  dialect  interference  or  a  failure  to  learn  tense  and 
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number.  Errors  of  syntax  could  be  divided  into  punctuation  errors, 
such  as  not  indicating  sentence  boundaries,  or  consolidation  errors, 
where  the  student  attempts  to  construct  complex  sentences  by  combining 
simple  ones,  but  fails  to  consolidate  appropriate  elements  or  ideas. 

Further  analysis  of  this  paper  provides  a  deeper  and  ' probably 
more  instructionally  useful  analysis  of  John's  writing.  Bartholomae 
has  taped  students  reading  their  own  papers;  these  tapes  reveal  the 
variation  between  writing  on  the  page  and  writing  that  is  spoken. 

He  finds  that  students  often  substitute  correct  forms  for  the 
incorrect  forms  on  the  pages ,  even  though  they  are  unaware  that  they 
make  these  substitutions .  The  transcript  of  John's  reading  is 
reproduced  below  with  substitutions  and  corrections  underlined  and 
Bartholomae' s  comments  in  parenthesis. 

This  assignment  calls  on  choosing  one  of  my  incident  making 
a  last  draft  out  of  it.  I  found  this  very  difficult  because 
I  like  them  all  but  you  said  I  had  to  pick  one  so  the  Second 
incident  was  decidejl  on.  Because  (John  goes  back  and 
rereads,  connecting  up  the  subordinate  clause.)  So  the 
second  incident  was  decided  on  because  this  one  had  the  most 
important  insight  to  my  life  that  I  indeed  learned  from. 

This  insight  explains;  why  adulthood  meant  that  much  as  it 
dose  to  me  because  I  think  it  always  influence^  me  to  change 
and  my  outlook  on  certain  thing_s  like  my  point-of-view  I 
have  one  day  and  it  might  change 'the  next  week  on  the  same 
issue.  (John  goes  back  and  rereads,  beginning  with  "like  my 
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point  of  view,"  and  he  is  puzzled  but  he  makes  no  additional 
changes.)  So  in  these  few  words  I'm  going  to  write  about 
the  incident  now.  My  experience  took  place  because  of  the 
type  of  school  I  went  to  (John  had  written  "too.")  Let  me 
tell  you  about  the  situation  (John  comes  to  a  full  stop.) 
first  of  all  what  happened  was  that  I  got  suspended  from 
school  (no  full  stop)  for  things^  that  I  felt  was  out  of  my 
control  sometime,  but  it  taught  me  a  lot  about 
responsibility  of  a  growing  man.  The  school  suspended  me 
for  being  late  ten  times.  I  had  accumulated  (for 
"accumulate")  ten  demerits  (for  "dementic")  and  had  to  bring 
my  mother  to  school  to  talk  to  a  conselor  and  the  Principal 
of  the  school  (full  stop)  what  went  on  at  the  meeting  took 
me  out  mentally  (full  stop)  period  (with  brio). 

In  this  admittedly  extreme  case,  the  writer  corrects  almost  every 
error  as  he  reads  the  paper  even  though  he  is  not  able  to  recognize 
that  there  are  errors  or  that  he  has  corrected  them.  For  example, 
when  asked  to  reread  sentences  and  notice  any  differences,  John  could 
not  see  the  error  in  "frew"  or  "dementic"  or  any  of  the  other  verb 
errors ,  and  yet  he  spoke  the  correct  form  of  most  every  verb  and 
corrected  every  plural.  On  the  basis  of  such  evidence,  Bartholomae 
argues  that  the  problem  for  students  like  John  appears  to  be  one  of 
performance  or  knowledge  of  the  procedural  rules  of  writing,  not  of 
linguistic  competence. 

Bartholomae' s  comparisons  of  written  and  spoken  text  emphasize 
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the  difficulty  students  can  have  in  editing  their  writing.  Failure  to 
correct  a  paper  is  not  necessarily  evidence  of  inattention  or  language 
disability.  Rather  failure  may  result  from  the  student  having  reached 
a  stage  of  fluency  in  writing  where  he  directly  accesses  a  written 
form  of  meaningful  words  (such  as  "dementic")  stored  in  memory  and 
does  not  utilize  the  slower  procedure  of  translating  speech  into 
writing . 

Research  of  this  kind  on  error  analysis  can  be  of  great  value  for 
the  composition  teacher.  It  provides  a  perspective  on  the  performance 
of  the  learner  that  is  interpretable  into  teaching  practice.  Both  the 
teacher  and  student  can  see  a  sensible  acceptance  of  the  writer's 
performance  and  clear  indications  of  the  differences  between  current 
performance  and  a  form  of  language  competence  acceptable  to  the 
teacher  and  the  student.  The  student's  difficulty  is  understood  as  a 
level  of  procedural  skill  that  requires  analysis,  understanding,  and 
practice . 

With  respect  to  the  task  of  the  assessment  and  diagnosis  of 
student  performance,  it  points  out  the  possibility  of  assessing  a 
learner's  "intermediate  system,"  his  current  rules  for  producing  the 
meaning  and  intent  of  the  writer  through  the  writing  code.  As  we 
refine  our  understanding  of  these  performance-based  errors,  we  can 

better  assess  the  error  in  written  composition  that  is  based  upon 

«• 

learned  and  repairable  procedural  knowledge  and  take  account  of  this 


in  instruction. 


The  work  on  composition- is  illustrative  of  the  point  I  wish  to 
make  of  the  necessary  interrelationships  between  the  analytical 
assessment  of  performance  and  effective  instruction.  At  the  present 
time,  a  number  of  institutions  are  using  writing  tests  of  some  form  to 
classify  their  students  into  instructional  groups.  The  intent  of 
research  on  writing  is  to  analyze  performance  so  that  the  level  at 
which  a  student  is  performing  .  can  be  specified  well  enough  for 
effective  guidance.  Classification  of  this  kind  is  not  seen  as  an 
evil  form  of  tracking.  Such  an  interpretation  is  prevented  by  the 
assumption  behind  the  tests:  They  are  a  means  of  identifying  the 
intelligent  systematicity  of  learners  that  they  and  the  teacher 
understand  can  be  used  for  the  further  improvement  of  skill. 

Rule  assessment  in  knowledge  of  scientific  concepts .  Methodology 
for  assessing  the  underlying  rule- structure  of  performance  and  the 
progressive  development  of  performance  complexity  in  children  has  been 
carried  out  by  Robert  Siegler  (Siegler,  1976,  1978).  In  the  course  of 
his  experiments  on  the  acquisition  of  knowledge  of  scientific 
concepts,  Siegler  has  formulated  an  approach  to  assessing  children's 
knowledge.  The  "rule  assessment  approach,"  as  he  terms  it,  is  based 
on  two  assumptions.  First,  that  human  reasoning  is  rule  governed, 
with  the  rules  progressing  from  less  sophisticated  to  more 
sophisticated  as  a  function  of  age  and  learning.  Second,  that  a 
useful  way  of  assessing  these  rule  progressions  is  to  design 
diagnostic  sets  of  problems  that  yield  distinct  performance  patterns. 
This  approach,  like  the  arithmetic  and  writing  work  that  I  have 
described,  can  determine  what  rules  an  individual  uses  in  performing  a 


Cask  as  well  as  determining  what  rules  are  common  co  various  groups  of 
Individuals  and  age  groups. 

The  way  Che  rule  assessment  procedure  works  can  be  illustrated  by 
using  a  variant  of  the  Inhelder  and  Piaget  (1958)  balance  scale  task 
that  was  used  in  Siegler' s  early  experiments.  The  balance  scale  task 
is  applicable  over  a  wide  age  range;  five-year  olds  may  know  that 
balances  such  as  teeter-totters  fall  to  one  side  when  they  have  more 
weight  on  one  end;  adults  however,  may  often  not  know  the  formal 
rules  determining  the  balance's  behavior.  The  first  step  in  the  rule 
assessment  approach  is  to  analyze  an  understanding  of  the  concept. 
The  balance  scale's  actions  are  governed  by  the  physical  concept  of 
torque:  The  scale  will  dip  in  the  direction  of  the  greater  of  the  two 
torques  acting  on  the  two  arms.  On  a  balance  beam,  pegs  are  placed  at 
equal  intervals  from  the  fulcrum  and  metal  discs  of  equal  weight  can 
be  hung  on  each  peg.  Torque  can  be  computed  by  multiplying  the  number 
of  weights  on  a  peg  by  the  ordinal  position  of  the  peg  or  the  distance 
of  the  peg  from  the  fulcrum.  The  side  having  the  greater  sum  of  the 
products  will  go  down;  if  the  products  are  equal  the  arm  will 
balance . 

Four  rules  describe  how  children  perform  the  balance  scale  task. 
Children  using  Rule  I  consider  only  a  single  dimension — the  amount  of 
weight  on  each  side.  Children  using  Rule  II  proceed  using  Rule  I  if 
the  amounts  of  weight  are  unequal;  but,  if  the  amounts  are  equal, 
they  also  consider  the  distance  of  the  weight  from  the  fulcrum.  A 
child  using  Rule  III  considers  both  the  amount  of  weight  and  the 


distance,  but  if  one  side  has  more  weight  and -the  other  has  its  weight 
farther  from  the  fulcrum,  the  child  guesses.  Finally,  Rule  IV 
children  compute  the  torques  when  necessary  and  choose  the  side  with 
the  greater  value. 

Knowledge  of  the  balance  scale  concept  is  assessed  by 
presentation  of  six  problem- types  (see  Figure  2).  Three  of  these 
problem  types  can  be  solved  without  computing  torques.  They  are: 
balance  problems,  with  equal  amounts  of  weight  equidistant  from  the 
fulcrum;  weight  problems,  with  different  amounts  of  weight 
equidistant  from  the  fulcrum;  and  distance  problems,  with  equal 
amounts  of  weights  different  distances  from  the  fulcrum.  The  other 
three  types  of  problems  Siegler  labels  conflict  problems,  because  one 
side  has  more  weight  and  the  other  more  distance,  thus  producing  a 
possible  conflict.  This  combination  of  rules  and  problem-types  makes 
it  possible  to  predict  the  particular  errors  that  children  make  as 
well  as  their  percentage  of  correct  answers .  For  example ,  on  distance 
problems,  children  using  Rule  I  should  always  predict  "balance"  rather 
than  choosing  the  side  with  the  weights  closer  to  the  fulcrum.  On 
conflict-distance  problems,  they  should  choose  the  side  with  more 
weight  rather  than  the  one  with  more  distance.  Siegler  has  carried 
out  such  assessment  for  various  types  of  scientific  concepts,  and  uses 
the  information  obtained  to  study  developmental  progression  and  to 
devise  instruction  that  facilitates  the  acquisition  of  higher  level 
rules  in  children  whose  existing  knowledge  is  at  a  lower  level. 


Balance 


Conflict-Weight 


Figure  2.  Problem  types  for  balance  scale  task  (from  Siegler,  1976). 
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Rule  assessment  approaches,  such  as  procedural  analysis  of 
arithmetic,  error  analysis  in  writing,  and  performance  rules  in 
scientific  understanding,  suggest  possibilities  for  diagnostic 
procedures  that  have  the  capability  of  providing  a  rather  deep 
analysis  of  the  general  understanding  of  a  subject  matter  that  an 
individual  brings  to  test  performance.  Test  items  developed  in  this 
way  could  provide  an  interesting  index  not  only  of  items  correct  or 
incorrect,  but  a  pattern  of  responding  that  yields  an  index  of  the 
level  of  a  person's  knowledge.  There  are  other  interesting 
psychometric  problems  in  a  procedure  of  this  kind.  For  example,  how 
many  items  corresponding  to  a  particular  problem  type  need  to  be 
presented  for  a  reliable  indication  of  performance?  To  what  extent  is 
there  a  consistent  progression  of  patterns  on  different  problem  types 
that  indicate  increasing  levels  of  knowledge  from  lower  to  higher 
order  rules?  Reliable  patterns  of  response  would  constitute  strong 
evidence  for  knowledge  advancement  as  a  result  of  instruction.  In 
assessing  creativity,  alternative  rules  generated  by  an  individual 
that  are  quite  logical,  but  do  not  correspond  to  the  specified 
progression,  could  be  ascertained. 

Bringing  the  concept  of  diagnosing  performance  regularities  at 
different  levels  of  learning  into  the  domain  of  psychometrics 
continues  my  theme  of  an  integration  between  testing  theory,  teaching 
practice,  and  knowledge  of  human  cognition.  Rule  assessment 
approaches  assume  that  conceptual  development  can  be  thought  of  as  an 
ordered  sequence  of  learned,  partial  understandings.  Because 
individuals  acquire  concepts  in  subject  matter  domains  to  various 
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levels  of  understanding  and  progress  through  the  development  of 
understanding  In  a  reasonably  predictable  fashion,  the  assessment  of 
levels  of  knowledge  can  be  linked  to  appropriate  Instructional 
decisions.  This  of  course  is  Hunt's  problem  of  the  match  (1961),  and 
a  problem  addressed  in  various  attempts  to  adapt  instruction  to 
individual  differences  (Glaser,  1977;  Glaser  &  Nitko,  1971). 

The  Nature  and  Assessment  of  Competence 

Over  the  past  ten  to  fifteen  years,  developments  in  cognitive 
psychology,  artificial  intelligence,  and  language  understanding  have 
spurred  increasing  investigation  of  the  characteristics  of  high  levels 
of  competence.  What  is  it  that  develops  through  years  of  learning  and 
experience  that  results  in  expertise  in  complex  skills  and  in  the 
amazing  efficiency,  judgment,  and  competence  shown  by  individuals  who 
are  very  good  at  what  they  do? 

In  past  studies  of  learning  and  skill  acquisition,  too  much 
effort  may  have  been  placed  on  analyzing  the  prerequisites  for 
learning  without  sufficient  information  about  the  nature  of  competent 
performance.  Emphasis  is  now  being  placed  on  understanding  the 
cognitive  structures  of  the  skilled  performer,  and  analyzing  the 
processes  involved  in  the  transformation  of  a  novice  into  an  expert. 
As  we  gain  increased  understanding  of  the  nature  of  high  levels  of 
competence  and  the  kinds  of  learning  and  experiences  that  foster  its 
development,  then  we  should  see  advances  in  techniques  of  instruction 
and  in  techniques  for  assessing  the  attainment  of  various  levels  of 


competence . 
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Understanding  expertise .  We  know  little  about  what  it  is  that 
the  expert  acquires  that  produces  his  or  ■ her  high  level  of 
performance,  and  investigation  often  results  in  surprising  and 
nonintuitive  findings.  This  work  requires  detailed  and  ingenious 
experimentation  because  skillful  performers  appear  to  observe  a  set  of 
rules  that  may  not  be  known  as  such  to  the  person  following  them. 
Before  looking  at  some  of  the  experimental  work,  an  antecdote  by  a 
student  of  writing  skill  makes  this  point  (Shaughnessy,  1977b).  She 
writes : 

In  my  few  attempts  to  work  contrastively  with  experienced 
and  inexperienced  academic  writers  on  the  same  assignments 
in  order  to  discover  hidden  features  of  competency,  I  have 
been  surprised  by  the  emergence  of  certain  skills  and 
orientations  I  had  not  thought  to  isolate  or  emphasize  as 
subjects  of  instruction.  I  have  noted,  for  example,  that 
the  craft  of  writing  has  a  larger  measure  of  craftiness  in 
it  than  our  instruction  seems  to  suggest.  Experienced 
academic  writers,  for  example,  appear  to  spend  little  time 
deliberating  over  their  main  intent  in  answering  a  question 
or  developing  an  essay;  this  conviction  evidently  reaches 
them  through  some  subtle,  swift  process  of  assessment  and 
association  that  has  doubtless  been  highly  cultivated  after 
years  of  writing  in  academic  situations.  But  after  this 
recognition  of  intent,  t.  'te  ,  follows  a  relatively  long 
period  of  scheming  and  plotting  during  which  the  writer, 


often  with  great  cunning,  strives  to  present  his  or  her 

intent,  (p.  319) 

Modern  seminal  work  in  the  study  of  expertise  began  with  the 
study  of  chess  skill  (Chase  &  Simon,  1973;  de  Groot,  1965,  1966; 
Simon  i  Chase,  1973),  work  familiar  to  all  students  of  cognitive 
psychology.  The  striking  difference  between  chess  masters  and  weaker 
players  shows  up  when  they  are  exposed  to  an  experimental  test  task — a 
chess  position  that  is  presented  for  a  very  brief  interval  of  time, 
five  to  ten  seconds.  In  this  situation,  Masters  are  able  to  recall 
the  position  almost  perfectly  from  memory  and  this  remarkable  ability 
drops  off  very  rapidly  below  the  Master  level.  Of  particular  interest 
is  that  this  result  cannot  be  attributed  to  any  kind  of  superior 
short-term  memory  capacity  of  the  chess  master  because  when  chess 
pieces  are  placed  randomly  on  a  chess  board,  recall  is  equally  poor 
for  Masters  and  weaker  players,  indicating  that  chess  Masters  have  the 
same  short-term  memory  limitations  as  everyone  else. 

The  series  of  experiments  by  Chase  and  Simon  indicate  that  the 
chess  Master  has  developed  a  large  repertoire  of  specific  patterns 
that  can  can  be  accessed  in  memory  and  quickly  recognized.  It  is 
estimated  that  the  size  of  the  Masters's  pattern  vocabulary  is  roughly 
50,000  different  configurations.  In  contrast,  a  good  club  player 
seems  to  have  a  recognition  vocabulary  of  1,000  patterns,  and  a  novice 
has  learned  to  recognize  very  few  patterns .  As  Chase  and  Simon  point 
out,  given  the  tens  of  thousands  of  hours  spent  looking  at  chess 
positions,  this  seems  comparable  to  the  fact  that,  with  similar  levels 
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of  pTactice,  good  readers  build  up  comparable  recognition  vocabularies 
for  words. 

Thus,  the  emphasis  in  understanding  chess  expertise  is  placed  on 
rapid  recognition  processes  that  tap  acquired  knowledge,  and 
high-level  competence  in  this  case  does  not  appear  to  reside  in 
conscious  analytical  thinking  processes.  The  chess  Master  is  a 
superior  recognizer  rather  than  a  deeper  thinker.  This  theoretical 
view  appears  to  explain  several  feats  of  expert  chess  players.  For 
example , 

"it  explains  how  a  chess  Master  is  able  to  defeat  dozens  of 
weaker  players  in  simultaneous  play:  because  for  the  most 
part  he  simply  relies  on  his  pattern  recognition 
abilities — his  so  called  'chess  intuition' — to  generate 
potentially  good  moves'*  (Chase  &  Chi,  1981). 

Findings  analogous  to  those  found  in  the  chess  work  are  beginning 
to  be  found  in  subject  matter  domains  taught  in  formal  schooling.  In 
solving  problems  of  the  kind  taught  in  elementary  physics, 
investigators  have  discussed  the  phenomenon  of  "physical  intuition," 
much  like  the  chess  Master's  "chess  intuition"  (Larkin,  McDermott, 
Simon,  &  Simon,  1980;  Simon  &  Simon,  1978).  Intuition  is  defined 
here  as  the  capacity  of  the  knowledgeable  physicist  to  solve  difficult 
problems  rapidly,  without  much  conscious  deliberation,  much  like  the 
□onanalytical  nature  of  the  chess  Master's  ability  to  find  good  new 
moves  associated  with  a  recognized  pattern.  In  studies  of 
expert-novice  differences  in  physics  problem  solving  carried  out  with 


my  colleagues  (Chi,  Feltovich,  &  Glaser,  In  press)we  have  found  chat 
the  expert's  knowledge  Is  organized  around  central  physics  principles 
whereas  the  knowledge  of  the  novice  Is  organized  around  the  physical 
entities  or  objects  directly  indicated  in  the  problem  description. 
Both  expert  and  novice  solve  the  problem,  but  the  way  the  problem  is 
Initially  represented  determines  different  problem  solving  procedures 
that  result  in  differences  in  efficiency  and  in  ability  to  handle 
difficult  situations.  Similar  results  are  obtained  with  architects, 
electronic  technicians,  and  political  scientists  in  the  way  they 
recall  architectual  plans,  circuit  diagrams  and  relevant  political 
information  (Akin,  1980;  Egan  &  Schwartz,  1979;  Voss  &  Tyler,  Note 
1).  Much  of  this  work  is  reviewed  in  a  report  by  Chase  and  Chi 
(1981). 

In  general  then,  the  learning  and  experience  of  the  competent 
individual  results  in  knowledge,  and  in  an  organization  of  that 
knowledge  into  a  fast  access  pattern'  recognition  or  encoding  system 
that  greatly  reduces  the  mental  processing  load;  understanding 
appears  to  result  from  these  required  knowledge  patterns  that  enable 
an  individual  to  form  a  particular  representation  of  a  problem 
situation.  Novices  have  systematic  knowledge  structures  at  a 
qualitatively  different  level  than  do  experts,  and  the  relative 
adequacy  of  the  initial  representation  of  a  situation,  that  acquired 
knowledge  structures  permit,  appears  to  be  an  index  of  developing 
competence . 

It  is  certainly  rash  to  jump  to  a  direct  extrapolation  from 
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current  experimental  tasks  to  assessment  tasks.  However,  the  seeds  of 
possible  new  procedures  are  apparent  (c.f.  Frederiksen,  Note  2).  As  a 
way  of  assessing  attained  competence,  will  it  be  possible  to  devise 
test  tasks  that  can  capture  the  level  of  knowledge  organization  that 
an  individual  brings  to  a  problem  situation?  Since  knowledge 
organization  appears  to  be  highly  correlated  with  effective  problem 
solving,  assessment  might  be  designed  to  identify  the  different  forms 
of  knowledge  representation  that  are  brought  to  bear  on  subject  matter 
situations.  As  we  develop  increased  understanding  of  the  bases  of  and 
growth  of  competence  in  various  domains  of  knowledge  and  skill,  we 
should  be  able  to  define  the  construct  validity  of  tests  in  terms  of 
the  concepts  of  theories  of  task  performance  rather  than  in  terms  of 
less  direct  correlational  evidence. 


Basic  skills  and  advanced  performance.  In  certain  investigations 
of  high-level  competence,  it  is  becoming  evident  that  human  ability  to 
perform  many  attention-demanding  tasks  is  rather  limited.  It  is 
probably  not  possible  for  two  attention-demanding  tasks  to  be  truly 
simultaneous  (Newell  &  Simon,  1972;  Shiffrin  &  Schneider,  1977).  As 
in  a  computer,  the  simulaneous  processing  of  two  tasks  can  be 
accomplished  by  time  sharing,  by  switching  attention  from  one  task  to 
another.  The  influence  of  this  fact  on  the  interaction  between  basic 
skills  and  advanced  components  of  performance  has  been  of  interest  in 
investigations  of  reading  comprehension  (Lesgold  &  Perfetti,  in 
press) .  Attention  may  alternate  between  the  basic  decoding  skills  of 
recognizing  words  and  the  higher-level  skills  of  comprehension  where 
sentence  ideas  are  integrated  into  memory. 
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This  oscillation  is  apparent  in  the  beginning  reader  who 
alternately  concentrates  on  sounding  out  a  word  and  then  on 
considering  what  the  word  means  in  the  context  of  what  is  being  read. 
While  these  component  processes  of  reading  may  work  well  when  tested 
separately,  they  may  not  be  efficient  enough  to  work  together.  Word 
decoding  processes  and  sentence  comprehension  processes  must  be  fast 
enough  to  avoid  desychronization.  Since  each  mental  process  takes 
time,  slowness  of  a  component  process  in  interaction  with  other 
processes  can  lead  to  a  breakdown  in  overall  proficiency  (Perfetti  & 
Lesgold,  1979).  The  notion  is  that  low  levels  of  reading  performance 
result  from  the  interfering  effects  of  slow,  inefficient  word  decoding 
on  the  execution  of  higher- level  comprehension  components. 
Less-skilled  readers  are  less  efficient  in  elementary  word  processing 
tasks;  this  takes  up  the  time  and  memory  space  that  is  necessary  for 
efficient  sentence  comprehension,  since  the  latter  depends  upon  the 
availability  of  relevant  knowledge  stored  in  memory  to  which  the  new 
information  can  be  related. 

This  general  hypothesis  of  the  interference  effects  between  basic 
and  higher-level  component  processes  raises  a  point  for  our 

consideration  here.  Apparently,  word  decoding  processes  need  to 

attain  a  certain  level  of  efficiency  before  more  advanced  processes 
.  can  be  carried  out.  Hence,  to  optimize  the  success  of  reading 

instruction,  it  may  be  useful  to  monitor  elementary  reading  practice 
to  ensure  that  it  attains  the  level  of  efficiency  required  for  it  to 
minimize  interfering  effects.  The  level  of  efficiency  of  word 

decoding  that  is  required  Is  a  research  question,  but  assessment  of 
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the  efficiency  levels  reached  by  readers  who  eventually  become  skilled 
might  provide  a  criterion-referenced  performance  index  of  whether  an 
individual  has  progressed  to  a  point  where  these  processes  are 
efficient  enough  to  facilitate  further  advancement.  In  general, 
empirical  and  theoretical  analyses  of  this  kind  might  suggest  ways  of 
measuring  competence  in  basic  skills  that  contributes  to  the 
attainment  of  advanced  levels  of  achievement  (Lesgold  &  Curtis,  in 
press) . 


Understanding  and  Improving  Aptitudes  for  Learning 

I  turn  now  to  the  third  item  on  my  agenda  for  research  and 
development  that  might  contribute  to  testing  in  the  future — the 
investigation  of  measures  of  aptitude  and  intelligence  that  correlate 
with  school  achievement.  Relevant  topics  here  include:  the  cognitive 
analysis  of  intelligence  and  aptitude,  discrepancy  between  actually 
assessed  and  potential  levels  of  intellectual  functioning,  and 
self-regulatory  skills  that  facilitate  learning  and  transfer. 


Analysis  of  measured  intelligence  and  aptitude .  In  a  recent 
review  of  the  measurement  of  intellectual  abilities,  John  Carroll 
(1978)  has  written  the  following: 

The  performances  required  on  many  types  of  mental  ability 
tests — tests  of  language  competence,  of  ability  to 
manipulate  abstract  concepts  and  relationships ,  of  ability 
to  apply  knowledge  to  the  solution  of  problems,  and  even  of 
the  ability  to  make  simple  and  rapid  comparisons  of 
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stimuli .. .have  great  and  obvious  resemblances  to 
performances  required  in  school  learning. .. .If  these 
performances  are  seen  as  based  on  learned,  developed 
abilities  of  a  rather  generalized  character,  it  would 
frequently  be  useful  to  assess  the  extent  to  which  an 
individual  had  acquired  these  abilities.  This  could  be  for 
the  purpose  of  determining  the  extent  to  which  these 
abilities  would  need  to  be  improved  to  prepare  the 
individual  for  further  experiences  or  learning  activities, 
or  of  determining  what  kinds  and  amounts  of  intervention 
might  be  required  to  effect  such  improvements.  These 
determinations.  However,  would  have  to  be  based  on  more 
exact  information  than  we  now  have.  (pp.  93-94) 

In  pursuit  of  this  goal,  the  past  eight  years  have  seen  the 
beginnings  of  substantial  activity  on  the  analysis  of  intelligence  and 
aptitude  test  performance  in  terms  of  the  methodology  and  concepts  of 
cognitive  science.  There  have  been  a  number  of  recent  reviews  and 
experimental  studies  on  the  work  being  done,  and  I  need  not  go  into 
detail  (Carroll,  1978;  Hunt,  1976;  Hunt,  Frost,  &  Lunneborg,  1973; 
Hunt  &  Lansman,  1975;  Pellegrino  &  Glaser,  1980;  Snow,  1980; 
Sternberg,  1977;  Whitely,  1976).  In  essence,  these  programs  of 
research  attempt  to  analyze  the  intellectual  functions  that  are 
assessed  in  measures  of  scholastic  aptitude. 

In  my  own  work  in  this  area  a  general  and  intimidatingly 
ambitious  scheme  that  has  guided  our  research  efforts  has  evolved  from 
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our  own  work  (Pellegrino  &  Glaser,  1979)  and  those  of  others.  It 
consists  of  a  series  of  stages  of  analysis.  Our  first  step  has  been 
to  identify  a  domain  of  tasks  associated  with  an  aptitude  factor, 
i.e.,  a  core  set  of  tasks  that  frequently  occur  across  widely  used 
tests  that  have  demonstrated  consistent  interrelationships  in 
factor-analytic  studies,  and  that  have  consistent  predictive  validity 
with  respect  to  a  criterion  performance  of  academic  achievement.  The 
second  step  is  to  develop  information  processing  models  for  the 
different  tasks.  These  models  involve  multiple  levels  of  cognitive 
processing — from  estimates  of  basic  memory  management  processes  to 
higher  level  strategies  controlling  the  integration  and  sequencing  of 
the  problem  solving  components  required  for  item  solution.  The  third 
step  is  to  use  the  models  of  task  performance  as  the  basis  for 
analyzing  item  difficulty  characteristics  and  individual  and 
developmental  differences  in  performance. 

Following  this  (and  some  years  down  the  road),  as  individual  and 
developmental  differences  in  cognitive  processes  used  on  aptitude  test 
tasks  are  identified,  then  similar  work  can  proceed  on  the  analysis  of 
criterion  tasks  used  to  establish  test  validities.  In  this  way,  the 
cognitive  performances  accounting  for  the  correlations  of  the  aptitude 
measure  with  criterion  performance  might  be  identified.  This  sets  the 
stage  for  constructing  an  information  processing  theory  of  validity. 
Based  on  such  a  theory,  investigation  can  proceed  on  the 
instructability  of  the  cognitive  components  of  aptitude  tasks  that 
facilitate  learning  of  academic  criterion  tasks. 


The  "zone  £f  potential  development In  Soviet  testing 
philosophy,  there  has  been  an  interest  in  the  development  of  reliable 
methods  for  the  diagnosis  of  learning  disabilities  (Brown  &  French, 
Note  3;  Vygotsky,  1978).  In  this  work,  a  distinction  is  made  between 
the  child's  actual  developmental  level,  i.e.,  the  level  of  development 
of  a  child's  mental  functioning  as  might  be  measured  on  standardized 
tests,  and  the  child's  level  of  potential  development,  i.e.,  the 
degree  of  mental  functioning  that  the  child  can  achieve  with  some  sort 
of  aid  and  assistance  in  the  course  of  testing.  Both  measures  are 
seen  as  essential  for  the  diagnosis  of  learning  ability  and  the 
concomitant  design  of  appropriate  instructional  or  remedial  programs. 
The  difference  between  these  measures,  or  the  "zone  of  potential 
development,"  or  "proximal  development"  as  Vygotsky  termed  it,  is 
conceived  of  as  an  indication  of  learning  potential.  Individuals  with 
the  same  score  on  a  mental  ability  test  may  vary  in  terms  of  their 
cognitive  potential.  The  assumption  is  that  a  major  difference 
between  average,  learning  disabled,  and  more  severely  retarded 
children  lies  in  the  width  of  this  zone.  This  is  seen  as:  "the 
distance  between  the  actual  developmental  level  as  determined  by 
Independent  problem  solving  and  the  level  of  potential  development  as 
determined  through  problem  solving  under  adult  guidance  or  in 
collaboration  with  more  capable  peers"  (Vygotsky,  1978). 

Vygotsky's  notions  about  the  relation  between  assessment  of  the 
zone  of  proximal  development  and  instruction  is  of  interest  here. 
Tests  can  be  used  either  to  determine  a  level  of  development  whose 
limits  instruction  should  not  exceed,  or  tests  can  assess  levels  of 


learning  chat  might  possibly  be  attained .  Accomplishing  the  latter 
could  orient  effective  instruction  toward  teaching  children  what  is 
lacking  so  far  in  their  development,  so  that  the  tested  level  is 
interpreted  as  a  possible  stepping-stone  to  a  higher  level.  Thus, 
learning  does  not  lag  behind  potential,  and  effective  instruction  is 
defined  as  that  which  is  in  advance  of  development  (Vygotsky,  1978). 

Self-regulatory  and  metacognitive  skills .  Metacognition  is 
defined  in  the  literature  in  a  number  of  ways,  but  I  consider  here 
that  aspect  of  it  which  refers  to  self- regulatory  and  self-management 
skills.  This  kind  of  skill  has  been  studied  in  detail  by  Brown  (1978) 
and  by  Belmont  and  Butterfield  (1978)  and  Meichenbaum  (1977). 
Regulatory  skills  refer  to  generalized  skills  for  approaching  problems 
and  for  monitoring  one's  performance.  These  skills  are  called 
metacognitive  because  they  are  not  specific  performances  or  strategies 
involved  in  solving  a  particular  problem  or  carrying  out  a  particular 
procedure;  rather,  they  refer  to  the  kind  of  knowledge  that  enables 
one  to  usefully  reflect  upon  and  observe  one's  own  performance.  As 
Flavell  (1976)  has  written: 

"Metacognition"  refers  to  one's  knowledge  concerning  one's 
own  cognitive  processes  and  products ... .For  example,  I  am 
engaging  in  metacognition. . .if  I  notice  that  I  am  having 
more  trouble  learning  A  than  B;  if  it  strikes  me  that  I 
should  double-check  C  before  accepting  it  as  a  fact;  if  it 
occurs  to  me  that  I  had  better  scruntinize  each  and  every 
alternative  in  any  multiple-choice  type  task  situation 
before  deciding  which  is  the  best  one;... if  I  sense  that  I 
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had  better  make  a  note  of  D  because  I  may  forget 
it ... .Metacognition  refers,  among  other  things,  to  the 
active  monitoring  and  consequent  regulation  and 
orchestration  of  these  processes  in... the  service  of  some 
concrete  goal  or  objective,  (p.  232) 

These  metacognitive  abilities  are  present  in  mature  learners  and 
take  on  the  characteristics  of  an  executive  control 
processor — introduced  as  an  overseer  in  many  current  models  of  memory. 
In  the  course  of  learning  and  problem  solving,  representative  kinds  of 
regulatory  performance  include:  knowing  when  or  what  one  knows  or 
does  not  know;  predicting  the  correctness  or  outcome  of  one's 
performance;  planning  ahead  and  efficiently  apportioning  one's 
cognitive  resources  and  one's  time;  and  checking  and  monitoring  the 
outcomes  of  one's  solution  or  attempt  to  learn.  As  Brown  ( 1 978 ) 
writes:  "These  forms  of  executive  decision  making  are  perhaps  the 
crux  of  efficient  problem  solving  because  the  use  of  an  appropriate 
piece  of  knowledge  or  routine  to  obtain  that  knowledge  at  the  right 
time  and  in  the  right  place  is  the  essence  of  intelligence"  (p.  82). 

Developmental  psychologists  have  indicated  that  these  regulatory 
skills  develop  with  maturity  a".<<  that  they  may  be  less  developed  in 
children  with  learning  disabilities  or  those  who  are  retarded.  It  is 
likely  that  these  skills  appear  in  various  forms  and  levels  of 
competence  over  a  wide  range  of  individuals.  The  especially 
interesting  characteristic  of  these  skills  is  that  they  may  be  the 
particular  aspect  of  performance  that  facilitates  transfer  from 
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learning  and  training  situations  to  new  situations.  Individuals  can 
be  taught  a  rule  or  procedure  that  improves  their  task  performance. 
But  if  transfer  to  new  situations  is  a  criterion  of  general  cognitive 
ability,  then  it  seems  important  to  teach  individuals  how  that  skill 
is  to  be  used  and  how  to  monitor  its  use. 

My  point  in  describing  self-regulatory  activities,  on  which 
active  research  is  being  pursued  at  the  present  time,  is  to  suggest 
them  as  important  candidates  for  assessment.  Tests  of  an  individual's 
skill  and  competence  in  these  self-regulatory  skills  might  be 
important  predictors  of  success  in  the  kind  of  problem-solving  ability 
that  produces  learning. 


Summary  and  Conclusions 

I  have  attempted  to  juxtapose  social-educational  requirements 
with  research  possibilities  that  are  more  and  less  remote  from  actual 
implementation.  This  juxtaposition  might  serve  a  useful  purpose 
because  sometimes  social  requirements  shape  research  questions,  and 
sometimes  research  findings  help  frame  the  attainable  form  of  a  social 
requirement . 

I  have  described  three  areas  of  social-educational  demand.  The 
first  is  the  shift  from  a  selective  educational  system  to  one  more 
designed  toward  helping  individuals  succeed  in  educational 
opportunities.  The  research  reported  indicates  new  possibilities  for 
diagnostic  measures  that  might  provide  information  helpful  for 


instructional  decision  and  guidance.  The  second  is  the  requirement 
for  improved  levels  of  literacy  and  problem  solving  ability  in  various 
domains  of  knowledge  and  skill. 

Research  on  the  differences  between  experts  and  novices  in 
knowledge  structure  and  cognitive  process  is  beginning  to  suggest 
techniques  for  assessing  the  attainment  of  various  levels  of 
competence  so  that  increasingly  higher  standards  of  skill  and 
understanding  might  be  achieved.  The  third  requirement  is  the 
understanding  of  individual  differences  in  measures  of  intelligence 
and  aptitude  in  order  to  study  the  extent  to  which  these  abilities  can 
be  improved  to  prepare  individuals  for  further  learning  and 
experience.  Research  has  begun  on  the  analysis  of  test  performance 
and  test  practice  in  terms  of  the  concepts  of  information  processing 
and  developmental  theory,  and  on  the  analysis  of  self-monitoring 
skills  in  cognitive  performance.  This  new  work  is  suggesting  the 
kinds  of  learning  skills  that  might  be  enhanced  through  educational 
and  other  environmental  interventions . 

Much  of  the  research  that  has  been  described  is  in  early  stages , 
but,  seen  in  terms  of  the  important  social-educational  demands 
involved,  the  importance  of  these  research  needs  might  encourage  more 
resources  and  talent  in  these  areas . 

In  1962  and  1963,  (Glaser,  1963;  Glaser  &  Klaus,  1962)  I  worked 
on  problems  of  assessing  human  performance,  and  out  of  this  grew  my 
concern  for  making  test  scores  informative  about  behavior,  rather  than 


Page  40 


about  relative  performance  on  poorly  specified  dimensions.  The 
concept  of  criterion-referenced  measurement  was  conceived  to  encourage 
the  development  of  procedures  whereby  assessments  of  proficiency  could 
be  referenced  to  stages  along  progressions  of  increasing  competence. 
At  that  time,  a  weak  link  in  the  construction  of  proficiency  tests  was 
understanding  and  defining  the  performance  to  be  measured.  The  1962 
paper  pointed  out  that  systematic  techniques  needed  to  be  developed  to 
more  adequately  describe  the  components  of  proficiency  at  various 
levels  of  competence.  For  this  purpose,  contact  was  made  with  certain 
aspects  of  the  then  prevalent  theory  of  learning  and  instruction, 
since  criterion-referenced  measurement  is  dependent  upon  an 
understanding  of  the  components  of  human  performance. 

In  the  ensuing  years,  outstanding  work  that  has  significantly 
influenced  testing  concepts,  if  not  teaching,  has  been  carried  out  by 
such  Individuals  as  Hambleton  <1978),  Millman  (1973),  Popham  (1978), 
and  many  others  with  some  useful  comment  by  Glass  (1978)  on  the  issue 
of  standards  and  criteria.  In  recent  years,  there  have  been 
significant  changes  in  the  theories  behind  our  understanding  of 
competence  in  areas  of  complex  knowledge  and  skills.  And  in  large 
part ,  I  have  atttempted  here  to  look  at  our  new  understanding  and 
relate  it  to  possible  developments  in  testing  and  psychometrics. 

•> 

The  thesis  of  Anastasi'a  presidential  address  to  this  division  in 
1966  (reminiscent  of  Cronbach's  address  in  1957)  was  that 
psychological  testing  should  be  brought  into  closer  contact  with  other 
areas  of  psychology  (Anastasi,  1967).  Increasing  specialization,  she 


said,  "has  led  to  a  concentration  upon  techniques  for  test 
construction  without  sufficient  consideration  of  psychological 
research  for  the  interpretation  of  test  scores”  (p.  305).  In  1977, 
Green  in  his  presidential  address  concluded  that  it  would  do  no  good 
to  use  tests  as  scapegoats  for  problems  that  demand  social  action 
(Green,  1978).  In  this  paper,  I  have  tried  to  combine  the  wisdom  of 
these  individuals  by  presenting  areas  of  social  concern  where 
education  and  testing  might  profit  from  coordination  with  potentially 
helpful  areas  of  psychological  research. 
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San  Diego,  CA  92152 

1  Ted  M.  I.  Yellen 

Technical  Information  Office,  Code  201 
NAVY  PERSONNEL  R&D  CENTER 
SAN  DIEGO,  CA  92152 

1  Library,  Code  P201L 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

1  Technical  Director 

Navy  Personnel  R&D  Center 
San  Diego,  CA  92152 

6  Commanding  Officer 

Naval  Research  Laboratory 
Code  2627 

Washington,  DC  20390 

1  Psychologist 

0NR  Branch  Office 
Bldg  114,  Section  D 
666  Summer  Street 
Boston,  MA  02210 


' 


University  of  Pittsburgh/Glaser  February  2,  1981 


Page  2 


Navy  Navy 


1  Psychologist 

ONR  Branch  Office 
536  S.  Clark  Street 
Chicago,  IL  60605 

1  Office  of  Naval  Research 
Code  437 
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1  Roger  Weis3inger-Baylon 
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San  Diego,  CA  92152 

Mr  John  H.  Wolfe 
Code  P310 
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University  of  Washington 
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Department  of  Psychology 
University  of  California 
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217  Stone  Hall 
Cornell  University 
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Dr .  Daniel  Gopher 
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UNIVERSITY  OF  PITTSBURGH 
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Dr.  Harold  Hawkins 
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University  of  Oregon 
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1  Dr.  Steven  W.  Keele 
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University  of  Oregon 
Eugene,  OR  97403 

1  Dr.  Walter  Kintsch 

Department  of  Psychology 
University  of  Colorado 
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1  Dr.  David  Kieras 

Department  of  Psychology 
University  of  Arizona 
Tuscon,  AZ  85721 

1  Dr.  Kenneth  A.  Klivington 
Program  Officer 
Alfred  P.  Sloan  Foundation 
630  Fifth  Avenue 
New  York,  NY  10111 

1  Dr.  Stephen  Kosslyn 
Harvard  University 
Department  of  Psychology 
33  Kirkland  Street 
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1  Dr .  Jill  Larkin 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

1  Dr.  Alan  Lesgold 
Learning  RAD  Center 
University  of  Pittsburgh 
Pittsburgh,  PA  15260 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  61801 

1  Dr.  Robert  A.  Levit 

Director,  Behavioral  Sciences 
The  BDM  Corporation 
7915  Jones  Branch  Drive 
McClean,  VA  22101 
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Faculteit  Sociale  Wetenschappen 
Ri jksuniversiteit  Groningen 
Oude  Boter ingestraat 
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NETHERLANDS 

1  Dr.  Erik  McWilliams 

Science  Education  Dev.  and  Research 
National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Mark  Miller 

Computer  Science  Laboratory 
Texas  Instruments,  Inc. 

Mail  Station  371,  P.0.  Box  225936 
Dallas,  TX  75265 

1  Dr .  Allen  Munro 

Behavioral  Technology  Laboratories 
1845  Elena  Ave.,  Fourth  Floor 
Redondo  Beach,  CA  90277 

1  Dr .  Donald  A  Norman 

Dept,  of  Psychology  C-009 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 


1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
400  Army  Navy  Drive 
Arlington,  VA  22202 

1  Dr.  Seymour  A.  Papert 

Massachusetts  Institute  of  Technology 
Artificial  Intelligence  Lab 
545  Technology  Square 
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1  Dr.  James  A.  Paulson 

Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

1  MR.  LUIGI  PETRULLO 

2431  N.  EDGEWOOD  STREET 
ARLINGTON,  VA  22207 

1  Dr.  Martha  Poison 

Department  of  Psychology 

University  of  Colorado 
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1  DR.  PETER  POLSON 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  COLORADO 
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University  of  Denver 

Denver, CO  80208 
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BUNDESMINISTERIUM  DER  VERTEIDIGUNG 

P0STFACH  1328 
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University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 
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University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

1  Dr.  Ernst  Z.  Rothkopf 
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Murray  Hill,  NJ  07974 

1  DR.  WALTER  SCHNEIDER 
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UNIVERSITY  OF  ILLINOIS 
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Clinton,  NY  13323 
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%  Dr.  Lonnie  R.  Sherrod 
Social  Science  Research  Council 
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1  Robert  S.  Siegler 
Associate  Professor 
Carnegie-Mellon  University 
Department  of  Psychology 
Schenley  Park 
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1  Dr.  Edward  E.  Smith 

Bolt  Beranek  &  Newman,  Inc. 

50  Moulton  Street 
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1  Dr.  Robert  Smith 

Department  of  Computer  Science 
Rutgers  University 
New  Brunswick,  NJ  08903 

1  Dr .  Richard  Snow 
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Stanford  University 
Stanford ,  CA  94305 

1  Dr .  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 

1  DR.  ALBERT  STEVENS 

BOLT  BERANEK  &  NEWMAN,  INC. 

50  MOULTON  STREET 
CAMBRIDGE,  MA  02138 

1  David  E.  Stone,  Ph.D. 

Hazeltine  Corporation 
7680  Old  Springhouse  Road 
McLean,  VA  22102 

1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  MATHEMATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD,  CA  94305 

1  Dr.  Klkumi  Tatsuoka 

Computer  Based  Education  Research 
Laboratory 

252  Engineering  Research  Laboratory 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr .  John  Thomas 
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1  Dr.  Douglas  Towne 
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1845  S.  Elena  Ave. 

Redondo  Beach,  CA  90277 
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1  Dr.  J.  Uhlaner 

Perceptronics,  Inc. 

6271  Variel  Avenue 
Woodland  Hills,  CA  91364 

1  Dr.  Benton  J.  Underwood 
Dept,  of  Psychology 
Northwestern  University 
Evanston,  IL  60201 

1  Dr.  Phyllis  Weaver 

Graduate  School  of  Education 
Harvard  University 
200  Larsen  Hall,  Appian  Way 
Cambridge,  MA  02138 

1  Dr.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455 

1  DR.  GERSHON  WELTMAN 
PERCEPTRONICS  INC. 

6271  VARIEL  AVE. 

WOODLAND  HILLS,  CA  91367 

1  Dr.  Keith  T.  Wescourt 

Information  Sciences  Dept. 
The  Rand  Corporation 
1700  Main  St. 


